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How do the various ideas currently used 
conceptualizing and modeling evaluation relate to the administrator? 
How can he best use them in guiding his institution in evaluation? 
This paper summarizes some of the major trends of thought related to 
evaluation and explores them from an administrative rather than from 
a research or instructional point of view« It presents a framework 
and discusses some of the issues involved in program evaluation in 
educational institutions. (Author) 
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A new type of evaluation, program evaluation f is beginning to take 
shape in education literature. It hasn't yet emerged as distinct and fully 
described phenomena. Whether it is a sub-category within educational 



evaluation or whether it is a parallel category is not. yet clear. What 
is abundantly clear, however, is that many of the newer conceptualizations 
of evaluation are not talking about the same things. For example. Stuff lebeam 
(1968) is discussing quite different evaluation than is Scriven (1972). 

Although new to the literature and perhaps not present in practioner 
jargon, program evaluation is not new in the actual practice of education. 
Program evaluation comprises a major part of the work of principals, 
superintendents, curriculum specialists, department chairmen and team 
leaders. It is an important element in the work of teachers who have 
choice in terms of what they emphasize and how they allocate resources 
among aspects taught. In keeping with the 15 minute time limit, this is 

r '* 

^»=4 a condensed version of the original paper. Main points are listed but 

not developed. The objectives of the study and paper are: 

*To briefly summarize some of the key ideas in recent writings 
on program evaluation which may be relevant to the administrator. 

*To present a concept of program evaluation in an administrative 
as opposed to an instructional or research context. 

*To identify issues related to further developing and operationalizing 
an administrative concept of evaluation. 
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Experiences contributing to this paper include: 

*Search of 'the literature for the past six years as input into a 
graduate level course on evaluation of Extension programs. 

*Interaction with students; interaction with progranuners and adminis- 
trators in Extension* 

*Participation in .the AERA workshop last fall on alternative conceptual- 
izations of evaluation. 

*Preparation of a manuscript for Syracuse ERIC in which I attempted 

^ to summarize ap ^ rlflsfilfy r^nxmj^^-framtffGnrkfi or approaches to evaluation 

(Steele, 1973). 

Limitations in the paper's usefulness to you: 

*My field is adult education. I have no contact with elementary or 
secondary school systems. Therefore my perceptions of how these 
ideas fit your field may not be accurate. 

*I am not an administrator, nor do I teach and research administration. 
Therefore, this paper comes not from within your own group, but from 
an outside observer. 

*The paper is a piecing together of things which others may see as 
unrelated to each other. It is more of a synthesis than it is an 
analysis of others* views of program evaluation. 

The paper includes the following main parts: 

*The pieces which are fitting together to create a concept of program 
evaluation. 

*A beginning attempt to define what program evaluation is and what 
it is not. 

*The relationship of program evaluation to administration. 

*Recommendations for further development and opera tionalization of 
program evaluation as an administrative concept. 

Factors and Ideas Contributing 
to a Concept of Program Evaluation 

Evaluation as a concept in education literature has experienced the 



following stages in development: 



*Evaluation was synonomous with measurement and was only concerned 
with evaluating student performance for grading and other purposes 
(Cuba, 1969). 

*Objectives became the focal point for measuring student performance. 

*Attainment of objectives then became the main means of evaluating 
the results of instruction and of curriculum (Tyler, 1950). 

*Attempts were made to include variables affecting results in models 
of evaluation (Taba and Sawin, 1962; Hammond, 1967). 



Considerable pressure was put on evaluation during the middle sixties 
and prevailing concepts were found wanting (Cuba, 1969; Glass, 1972). As a 
result more literature on evaluation has been produced and more different 
ideas about evaluation has been* presented in the past eight years than in 
the previous fifty years. 

Factors that have recently emerged or taken on new perspectives and 

that are relevant to the developing concept of program evaluation include: 

*The term pr^jgram is becoming more common in education as a result 
of its use in PPBS — program planning budgeting system — programmed 
instruction, and programs developed through R and D centers. 

^Theory and concepts of operational research and managerial science 
have spread to education. For example, the concepts of system and 
management of systems is more widespread. 

*With pressures for relevance and taxpayers' revolts, assessments of 
attitudes toward educational programs and their results are deemed 
as important parts of evaluation. 

^Distance between the source of funds and the local school (ie, the 
amount funneled to the federal treasury and then back to the local 
school system) has introduced a concept of accountability that 
goes beyond accountability to a local school board. 

*Whether considered within prior definitions of evaluation or not, the 
evaluative activities involved in aaaredition and on-site inspections 
by the State Department of Instruction have had to continue. 

* Administrative decisions became much more complex as school systems 
trebled in size over recent years. 



^Curriculum decisions have become much more complex with the "knowledge 
explosion" and the challenges of preparing young people to take their 
place in the complex and sophisticated America of the future. 
Educational decisions are much more difficult than ,they were thirty 
years ago* 

*Greater attention is being paid and there is raore conflict generated 
today in terms of the ultimate goals in education. 

*Instructional objectives and institutional goals^ the micro and the 
macro levels, have emerged as specialized fields within the literature. 
How the two relate to each other is less clear. More attention is 
beginning to be paid to hierarchies and networks of objectives and 
goals • 

Recent ideas about evaluation which have particular relevance to program 
evaluation include: 

*Evaluation as input into decision making. The purpose of evaluation 
is to improve rather than prove (Phi Delta Kappa, 1970). 

*Four types of decisions with corresponding types of evaluation — context, 
input, process and product (Ph Delta^ Kappa, 1971). 

"^Formative evaluation as different from summative evaluation (Scriven, 1967). 

*Evaluation*s contribution at all ctagro of program development (Provus, 
1971; Lindvall and Cox, 1970; Tripodk, Fellin, and Epstein, 1971). 

*The act of judgment as an essential part of evaluation (Stake, 1967). 

^Criteria as essential elements in evaluation (Herzog, 1959; Suchman, 
1967; Phi Delta Kappa, 1971). 

*Kinds of criteria categories to be used in examining programs. For 
example, adequacy of effort, effectiveness, efficiency (Suchman, 
1967; TriPodi, Fellin and Epstein, 1971). 

^Institutional evaluation as being something more than course evaluation 
(Forehand, 1970). 

*Reductionism — testing the pieces of the clock separately doesn^t mean 
that the clock will run (Hawkridge, 1970). 

*In addition to meeting traditional scientific criteria, validity, 
reliability and objectivity, evaluation should meet the practical 
criteria of relevance, importance, scope, creditability, timeliness, 
and pervasiveness (Phi Delta Kappa, 1971). 

*Evaluation should meet the prudential criteria of efficiency (Phi Delta 
Kappa, 1971). It must balance benefits with costs of evaluation. 
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*Similarities and differences in evaluation and research (Hemphill, 
1969; Glass and Worthen, 1972). 

*Judging the results of the program against the intitntinK need 
(Scriven, 1972), the mission of the unit and the intent of the 
particular kind of education, ie., developmental, problem solving 
mastery of content, etc. (Steele, 1972). 

*Program as a productive system with inputs, processes, and outputs 
(Alkin, 1967; Stake, 1967). 

*Evaluation as comparing more than one means of attaining the same end 
(Scriven, 1972; Stuff lebeam, 1972). 

* pg>v^«^nngjh:^ l ^itv f pr nrpnvnrnH4-flT^^Ti^ good stewardship of resources ^ 

used (Browder, 1971). 

*Consideration of both real and opportunity costs and of both private 
and social benefits in cost-benefit analysis. 

*Trade offs as an important and realistic concept in evaluation (Glass, 1972). 

*Importance of involving people in setting standards, interpreting 
data, and developing conclusions (Verduin, 1967; Byram and Robinson, 
1970; Rippey, 1972). 

*Evaluation as management methodology (Lange, 1970). 

*The relationship of education to social policy (Berlack, 1970) of 
evaluation and politics (Cohen, 1970; Eash, 1972) and of evaluation 
and social policy (Freeman and Sherwood, 1970; Provus, 1972) 

Few of the above concepts come out of a discussion of program evaluation 

per se, but each contributes to the piecing together of a concept of program 

evaluation. 



What is Program Evaluation? 

Definitions of evaluation that are particularly relevant in formulating 

a concept of program evaluation include: 

*Evaluation is the systematic process of judging the worth, desirability, 
effectiveness, or adequacy of something according to definite criteria 
and purposes. The judgement is based on a careful comparison of 
observation data with criteria standards. Precise definitions 
of what is to be appraised, clearly stated purposes, specific 
standards for the criteria traits, accurate observations and measure- 
ments, and logical conclusions are the hallmarks of valid evaluation 
(Harris, 1968). 



*Social program evaluation is the systematic accumulation of facts for 
providing information about the achievement of program requisites 
and goals relative to efforts, effectiveness, and efficiency within 
any stage of program development. The factors of evaluation may be 
obtained through a variety of relatively systematic techniques, and 
they are incorporated into some designated system of values for making 
decisions about social program (fripodi, Fellin and Epstein, 1971) • 

*Evaluation is quality control of the processes and outcomes of an 
educational program (Gottman and Clasen, 1972). 

*Evaluation is the process of using information and criteria in 
judging decision alternatives* Adaptation of Phi Delta Kappa, 1972* 

^ (This adaptation is made to take the defdt>i4;ixm-Jrom^^ 
the professor-researcher whose role is that of providing informa-' " 
tion and transfer it to the world of the administrator whose role 
is that of using evaluative information in making decisions.) 



Program evaluation is not: 

*The same thing as instructional evaluation. 

*Only examining the results of programs— attainment of objectives or 
other indicators of results. 

*The same thing as program research or evaluative research. 
*The same thing as accountability. 
*The same thing as systems analysis. 
Program evaluation can be described as: 

*Focusing on macro-levels but also being concerned with how each 
unit fits into a course, each course fits into a curricula, each curricula 
fits into the total program of a school, the program of each school 
fits into the total of life-long education. 

*Recognizing that good evaluation during formative stages results in 
a greater chance of formative evaluation showing expected outcomes. 

*A generic process including a variety of foci, strategies, and 
evaluation approaches and techniques. 

*Essentially being a process of forming judgements by using criteria 
and/or decision alternatives and evidence germane to the criteria or 
key discriminating factors. 

*Primarily concerned with improving decisions and improving education 
through facilitating better decisions. 

*Concerned with the extent to which the mission and institutional goals 
of the school system and of education are being met. 



*Concerned with the way in which resources (of the child, the teacher, 
support personnel, the school district, and society) are being used 
in the process of education. 
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*Concerned with underlying value questions and issues that affect 
educational decisions* 

*Focusing on education as a system made up of parts • 

*Concerned with the way that various parts of the educational system 
function together in facilitating education, 

♦Concerned with the quality and accuracy of program decisions throughout 
the total educational process. 

♦Concerned with quality and accuracy of choice among alternatives 
at the program level as to (a) who will receive what kinds of 

^education, (b) what will be taught, (c) how it will be taught, 
(d) what le vel Of pcjf o imanco will bg expected ■ — 

♦Recognizing that programs should be judged on more than one set of 
criteria. For example, suitability, quality, and efficiency need to 
be considered as well as effectiveness. 

♦Examining the addative effect of several courses in a curriculum 
area or several years of schooling. ^ 

♦The prerogative of those who are intimately involved with the program — 
teachers, administrators, students, parents, and the local community. 

♦Sometimes being facilitated by outside experts, but because the 
decisions lie within the school system, the bulk of the evaluation 
must be done within that system. 

♦Examining the relationship of the program to community and societal 
context. Is it relevant to real life needs? 

♦Concerned with totals and wholes rather than independent parts 
standing in isolation to each other. 

♦Drawing both on scientifically processed data and upon wisdom built 
through human experience, and knowing when each is appropriate. 

♦Identifying and dealing with the most imperative trade offs. 

♦Able to identify the difference between the best that is possible 
within a given situation and an ideal state. 

♦A dynamic, interactive, and continuous process. Although it can be 
facilitated by episodic studies of more formal nature, such studies 
don*t constitute the whole of evaluation. 

♦Comprised of a variety of evaluative activities which can aid in program 
evaluation and decision making? examination of results; use of advisory 
and steering committees, comprehensive reviews, experimental research 
management information systems, PERT, etc. 
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*Deallnf5 with political roalltlos both withfn and whlumi tho odurattnanl 
system. 

*Concerned with the people interactions necessary for decisions to be 
carried out* 

*Defining what constitutes "good" education and "successful" educators . 

*Establishing criteria on which education in total and the contributions 
of the various parts of the system are judged, 

*Dealing with questions and challenges_posed-by -cit±ZHil gtoupa auU- 

the gener^t pTibltc^ " 

*Concerned with the impact that participating in education has upon 
students, families, conmiunities , and society. 

*A function of administration and program management. It must be done 
within those roles. 

The above statements are bits and pieces seen in the mirror darkly. 

All deal with something that is evaluative and something that is more than 

the traditional concept of evaluation. 

How does one take a generic process like program evaluation which can 
encompass many activities and compress it into a short definition? As a 
start try the following. 

Program evaluation is the process by which^criteria and evidence are 
used in forming judgements about programs, alternative programs, and 
alternatives within programs and the programming system, as a means of 
facilitating key decisions about education. 

Program evaluation is a process by which evidence, criteria, and 
judgement are used in managing resources and facilitating accurate and 
appropriate decision making in areas of major importance about education. 
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These two definitions are posed as examples and as stimuli to you in 
formulating a definition of what program evaluation is as you view it 
within the administrative context. Such definitions are needed both by 
the field as a whole, and by you in your own situation. 

A Concept for Administrators 
Evaluation which emphasizes decision making and management of resources 
which goes beyond the decisions that control specific teacher-student inter- 
actions is an administrative rather than instructional or research concept* 
In this concept of evaluation the administrator is the key actor. He must: 

*Set the philosophy of program evaluation which prevails within the 
system and the climate of security within which it operates. 

*Identify the kinds of decisions that need to be made and the 
nature of the evaluations inherent in those decisions. 

*Establish criteria for distinguishing among alternatives and for 
judging the functioning and results of the educational system. 

*Know how to judge the limitations of evidence that is easily at hand. 

*Know when to invest resources in extensive evidenc^e collection. 

*Know how to use data once it is at hand. 

*Be able to understand and influence and utilize the evaluations 
made by others, school board members, public groups, etc. 

*Be able to effectively lead his staff in getting the kinds of 
evaluation done which improve decision making and as a result 
improve programs. 

Program evaluation has potential for being an important tool 

in day-to-day operations and in times of major debate, crises, or setting 

new directions. However, much yet needs to be done to shape the tool to 

practical application. 
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Issues and Needs for Further Development 

First, 

*Administrators need to decide whether the phenomena I have sketched 
and labeled as program evaluation actually is something that is 
real and important to them. This paper can be seen as 
presenting a hypothesis that needs to be tested. Perhaps adminis- 
trators don*t need any help with decision making or recognize that 
there is an evaluative component in decision making. 

If there is merit in pursuing a concept of program evaluation in education, 

then, ' 

• *AlFernative definitions olr"pifogr-<KiV evaluation need to be debated. 

*Component parts need to be identified. Currently some concepts 
describe elements within a system — personnel, facilities, etc; 
others focus on stages; others on decision categories. All probably 
fit well within the oncept of program evaluation, but how do they 
fit needs to be identified? 

*Parameters of the concept need to be tentatively set and then debated. 

Then, assuming that the definitions do emphasize the real-life dynamics of 

managing educational decisions, there may need to be further development of 

procedures for: 

^Describing and guiding how evaluation functions within the dynamics 
process of decision making. 

*Describing and improving the nature of human judgement. 

^Evaluating needs and setting priorities. 

^Judging the limitations of evidence and the consequences apt to 
be caused if the data are faulty. 

*Efficiently using data, or for realistically judging what scientifically 
handled data (research type studies, computer information system out- 
puts, etc.) are worth their cost. 

^Debating and selected criteria to be used in key decision situations; 
procedures for translating conceptual criteria into realistic guides. 

^Assessing the worth of conflicting values and positions taken based 
on those values. 

*Making processes of interacting wi<"h people in judging alternatives 
and reaching decisions more efficient and less frustrating. 



The items above are examples to stimulate your thinking. What evaluative inputs 
do administrators really need to help them with decision making? 

Challenge to Administrators 
I*m not sure that I'm talking to administrators. I have a hunch that 
you are probably much more apt to be either prof essoin and researchers or 
technical support staff. However, on the chance that there are some 
^"dminisTfatl>r^-Tiere7 'I would suggest that It is time that* you begin for- 
mulating concepts of program evaluation which meet the practical tests of 
time and of day-to-day activity. You have handed conceptualization and 
implementation of evaluation over to outside experts a bit too willingly. 
If you are going to get a tool that really works for you, you have to 
become a partner at the drawing board. You will have to take the leader- 
ship in promoting the development and improvement of concepts of program 
evaluation. 
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